{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "BN3AqkFOzaSM"
   },
   "source": [
    "# OmniSafe Tutorial - CLI Command\n",
    "\n",
    "OmniSafe: https://github.com/PKU-Alignment/omnisafe\n",
    "\n",
    "Documentation: https://omnisafe.readthedocs.io/en/latest/\n",
    "\n",
    "Safety-Gymnasium: https://www.safety-gymnasium.com/\n",
    "\n",
    "[Safety-Gymnasium](https://www.safety-gymnasium.com/) is a highly scalable and customizable Safe Reinforcement Learning library, aiming to deliver a good view of benchmarking Safe Reinforcement Learning (Safe RL) algorithms and a more standardized setting of environments. \n",
    "\n",
    "## Introduction\n",
    "\n",
    "在本节当中，将为您介绍OmniSafe的CLI工具，它可以让您完全脱离代码,通过命令行的命令调用OmniSafe，即使完全不懂编程知识，通过本节的介绍，也能轻易上手OmniSafe。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oDTmwX4d5jmR"
   },
   "outputs": [],
   "source": [
    "# Install from PyPI. if you already have OmniSafe installed, please ignore the code in this cell.\n",
    "!pip install omnisafe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AbGRUmvz5nGL"
   },
   "outputs": [],
   "source": [
    "# Install OmniSafe and dependencies from source\n",
    "## if you already have OmniSafe installed, please ignore the code in this cell.\n",
    "## Clone the repo\n",
    "!git clone https://github.com/PKU-Alignment/omnisafe\n",
    "%cd omnisafe\n",
    "\n",
    "## Install OmniSafe\n",
    "!pip install -e ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NHf5aG3d5qZS"
   },
   "source": [
    "## CLI Usage Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WFANT3Fr6wtm"
   },
   "source": [
    "### Quick experiment\n",
    "\n",
    "`train`命令旨在为您提供快速运行实验的接口。\n",
    "\n",
    "通过以下命令，您可以使用默认参数开始训练，并且可以通过命令行非常方便地修改一些常见的参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5tvIl0fS63Cl"
   },
   "outputs": [],
   "source": [
    "!omnisafe train --algo PPO --total-steps 2048"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QGbJe-ca88NM"
   },
   "source": [
    "如果您想进一步了解`train`命令的用法，只需要`--help`。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "A2wGFbyN9DYE"
   },
   "outputs": [],
   "source": [
    "!omnisafe train --help"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "YKukyW-g9dWm"
   },
   "source": [
    "对于不常见的参数，我们也支持修改，但这超出了`train`命令被设计用于快速简易实验的初衷，因此如果有修改大量不常见参数的需求，请参阅接下来的`train-config`部分。\n",
    "\n",
    "在下面的例子中，我们在命令行中修改了默认参数当中属于`algo_cfgs`的`steps_per_epoch`的值。\n",
    "\n",
    "注意：\n",
    "*   对于嵌套的参数键值，请使用`:`进行连接。\n",
    "*   要修改参数，对应的键值必须存在于事先定义的`yaml`文件当中。\n",
    "*   保存OmniSafe当中每个算法默认参数的`yaml`文件，请查阅[此处](https://github.com/PKU-Alignment/omnisafe/tree/main/omnisafe/configs)。\n",
    "*   要修改不常见的参数，请使用`--custom-cfgs`，其中，您必须将键值对按照对应关系给出，且每一个字符串前都需要给出`--custom-cfgs`标志。例如：`--custom-cfgs key1 --custom-cfgs value1 --custom-cfgs key2 --custom-cfgs value2`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aWugQFB_-FOj"
   },
   "outputs": [],
   "source": [
    "!omnisafe train --algo PPOLag --env-id SafetyHumanoidVelocity-v1 --total-steps 2048 --custom-cfgs algo_cfgs:steps_per_epoch --custom-cfgs 1024"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "A0oOSrioiWbv"
   },
   "source": [
    "### Configure your experiments\n",
    "`train-config`从您给定的yaml文件路径读取您给出的参数，在默认参数上增量更新。\n",
    "\n",
    "通过`train-config`命令，您可以给出更详尽更细致的参数，而不用在容易出错的命令行中应对海量参数。\n",
    "\n",
    "注意：\n",
    "\n",
    "\n",
    "*   您需要创建自己的参数文件，该文件需要遵循与默认参数一致的文件格式，对于库中支持的算法，其文件格式可参考[此处](https://github.com/PKU-Alignment/omnisafe/tree/main/omnisafe/configs)。\n",
    "*   配置文件可创建在任意位置，只需在命令行调用`train-config`命令时给出其路径即可，我们推荐您定义在当前运行命令的路径下，因为您运行命令所产生的文件也会保存在此。\n",
    "\n",
    "\n",
    "下面给出一个例子，我们将会以不同方式再次运行上一个实验。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "finO392PkJqi"
   },
   "outputs": [],
   "source": [
    "!mkdir ./save_config\n",
    "%cd ./save_config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fhmXec87l9lt"
   },
   "source": [
    "接下来，定义您的实验参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "D05bUldkmCit"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "echo \"algo:\n",
    "  PPOLag\n",
    "env_id:\n",
    "  SafetyAntVelocity-v1\n",
    "train_cfgs:\n",
    "  total_steps:\n",
    "    1024\n",
    "  vector_env_nums: 1\n",
    "algo_cfgs:\n",
    "  steps_per_epoch:\n",
    "    1024\" > config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I3I-D-zDpZsT"
   },
   "source": [
    "接着，只需要为`train-config`命令指出配置文件的目录即可。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "INH3AIZZpYzl"
   },
   "outputs": [],
   "source": [
    "!omnisafe train-config ./config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sQmvE0ZqqYGM"
   },
   "source": [
    "同理，通过`--help`您可以查看更多相关信息。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RGH4stQJqh3r"
   },
   "outputs": [],
   "source": [
    "!omnisafe train-config --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BANIxfB7qrE1"
   },
   "source": [
    "### Render and evaluate your experiments\n",
    "\n",
    "在CLI当中，我们也为可视化训练数据提供了支持，您可以通过`eval`命令调用。\n",
    "`eval`支持多参数灵活调用，通过`--help`可以获取详尽的解释。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "33FepsLErVmA"
   },
   "outputs": [],
   "source": [
    "!omnisafe eval --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VK1CFzbWr6Vl"
   },
   "source": [
    "接下来，我们对上一个运行的实验进行render和evaluate，可视化的结果会保存到实验数据的同目录下。\n",
    "\n",
    "注意：\n",
    "\n",
    "*   通过`--no-render`参数，可以关闭可视化。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jMJZZH43sgeb"
   },
   "outputs": [],
   "source": [
    "%ls\n",
    "%cd ./train_dict\n",
    "%ls"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "-Y9NJgUbt9rh"
   },
   "source": [
    "注意：在没有显示器的服务器上，render时可能会遇到麻烦，请尝试执行如下操作。若您在自己的机器上遇到问题，请通过GitHub为OmniSafe提出[issue](https://github.com/PKU-Alignment/omnisafe/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)来让我们协助解决。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oDIbJfEJuHpJ"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "apt-get install libosmesa6-dev\n",
    "apt-get install python3-opengl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 627,
     "status": "ok",
     "timestamp": 1680321048452,
     "user": {
      "displayName": "day may",
      "userId": "09325921245341195745"
     },
     "user_tz": -480
    },
    "id": "nPpSlZhb7oWw",
    "outputId": "7b7c077f-c32d-4963-dfa7-297bb6f90bd8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: MUJOCO_GL=osmesa\n",
      "env: PYOPENGL_PLATFORM=osmesa\n"
     ]
    }
   ],
   "source": [
    "%env MUJOCO_GL=osmesa\n",
    "%env PYOPENGL_PLATFORM=osmesa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "x4e9_eestRj4"
   },
   "outputs": [],
   "source": [
    "!omnisafe eval ./PPOLag-{SafetyAntVelocity-v1} --num-episode 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aQeHlYP18NFN"
   },
   "source": [
    "让我们检查一下生成的视频。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "V-2RBJqs8UvN"
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "from pathlib import Path\n",
    "\n",
    "from IPython import display as ipythondisplay\n",
    "\n",
    "\n",
    "def show_videos(video_path='', prefix=''):\n",
    "    \"\"\"\n",
    "    Taken from https://github.com/eleurent/highway-env\n",
    "\n",
    "    :param video_path: (str) Path to the folder containing videos\n",
    "    :param prefix: (str) Filter the video, showing only the only starting with this prefix\n",
    "    \"\"\"\n",
    "    html = []\n",
    "    for mp4 in Path(video_path).glob(\"{}*.mp4\".format(prefix)):\n",
    "        video_b64 = base64.b64encode(mp4.read_bytes())\n",
    "        html.append(\n",
    "            '''<video alt=\"{}\" autoplay \n",
    "                    loop controls style=\"height: 400px;\">\n",
    "                    <source src=\"data:video/mp4;base64,{}\" type=\"video/mp4\" />\n",
    "                </video>'''.format(\n",
    "                mp4, video_b64.decode('ascii')\n",
    "            )\n",
    "        )\n",
    "    ipythondisplay.display(ipythondisplay.HTML(data=\"<br>\".join(html)))\n",
    "\n",
    "\n",
    "# Please fill the path of folder containing your video which is shown above here\n",
    "show_videos(video_path='')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "bC1gRloPu-rF"
   },
   "source": [
    "### Quick benchmarking and large-scale experiment\n",
    "\n",
    "`experiment_grid`是一个非常好用的工具。通过它，您可以快速测试感兴趣的所有可行的参数组合对算法性能的影响。而通常，仅当需要指定的参数较多时才需要使用`experiment_grid`，因此您只能通过配置文件来调用它。配置文件的格式请参考[此处](https://github.com/PKU-Alignment/omnisafe/blob/main/examples/benchmarks/example_cli_benchmark_config.yaml)。\n",
    "\n",
    "简单来说，您需要指出参数的键值，若是嵌套键值，则使用`:`分隔，其所有可行的取值需要用中括号`[]`包裹，缩进遵循yaml文件的格式要求。\n",
    "\n",
    "注意：\n",
    "\n",
    "*   我们建议您将`num-pool`的值设定为可以整除总实验数的值，这样可以保证您的机器在整个实验过程中，同一时刻执行的任务数大致一样。\n",
    "*   `exp_name`指定了您本次运行的所有实验所在的目录，为了确保两次实验的文件不会产生冲突，在每一次运行`experiment_grid`的时候，您都需要显式给出其值。\n",
    "\n",
    "\n",
    "\n",
    "在OmniSafe的CLI工具当中，您可以通过`benchmark`命令调用`experiment_grid`。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Amjgyq2gc8Av"
   },
   "source": [
    "当然，您还是可以通过`--help`获取进一步的使用方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0vTStdXIdEPO"
   },
   "outputs": [],
   "source": [
    "!omnisafe benchmark --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SlponENqd_gi"
   },
   "source": [
    "接下来，让我们通过`benchmark`命令运行一组实验。当需要调整大量参数时，这会让您的工作变得相当高效。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "huSt8vCSfhtY"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "echo \"algo:\n",
    "  ['PolicyGradient', 'NaturalPG']\n",
    "env_id:\n",
    "  ['SafetyAntVelocity-v1']\n",
    "logger_cfgs:use_wandb:\n",
    "  [False]\n",
    "train_cfgs:vector_env_nums:\n",
    "  [2]\n",
    "train_cfgs:torch_threads:\n",
    "  [1]\n",
    "train_cfgs:total_steps:\n",
    "  4096\n",
    "algo_cfgs:steps_per_epoch:\n",
    "  2048\n",
    "seed:\n",
    "  [0]\" > grid_config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Qw5e3XsWf2z5"
   },
   "source": [
    "要运行命令，必须：\n",
    "\n",
    "1.   给出实验名称\n",
    "2.   给出本次实验同时运行进程的最大数目。\n",
    "3.   给出参数配置文件的路径。\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9p_KydAUf1lS"
   },
   "outputs": [],
   "source": [
    "!omnisafe benchmark test_grid 2 ./grid_config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f34rBm9Ug4ep"
   },
   "source": [
    "### Quick analyzing the results\n",
    "\n",
    "在`train`和`train-config`命令当中，您可以直接通过指定`--plot`, `--render`和`--evaluate`来选择是否在训练结束时绘制训练曲线图，可视化过程中的model以及测试model性能。\n",
    "\n",
    "而在`benchmark`当中，由于相比于单个实验，要进行分析的复杂性显著更大，我们引入了一个单独的`analyze-grid`命令实现对结果的分析。\n",
    "\n",
    "注意：\n",
    "\n",
    "\n",
    "*   `render`与`evaluate`的运行过程对于model来说并没有区别，都是将model在与训练环境实例不同的环境实例上进行测试，可以将render理解为同时渲染视频的evaluate。\n",
    "*   测试时，网络`deterministic==True`成立。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "imSYTMpKiyQP"
   },
   "source": [
    "接下来介绍`analyze-grid`命令。通过`--help`，您可以看到有如下参数可供指定：\n",
    "\n",
    "`path`：指出一组`experiment_grid`实验的保存路径，只要给出的文件夹下存在一个完整的由`experiment_grid`所启动的实验文件夹即可，程序会自动递归地寻找其具体位置。\n",
    "\n",
    "`parameter`：指出您本次想要通过对比性能分析其作用的参数名称。同上，若是嵌套参数请使用`:`分隔。\n",
    "\n",
    "`compare_num`：指出您在分析参数作用时，在同一张图上最多想看到几种不同的参数取值，OmniSafe会自动生成所有这样数目的对比组合，举例来说，假如当前有3个算法，其它每个参数只有一种取值，`parameter='algo' compare_num=2`，此时您将会得到3张图，其中每张图对比了两种算法在当前环境和参数上的性能。这可以很好地减少您对比时的心智负担。\n",
    "\n",
    "`cost_limit`：展示在性能图cost的部分当中的黑色线，提示应该遵循的cost_limit取值。\n",
    "\n",
    "您可能会注意到，在上一节当中用户还可以指定`values`选取特定的想要对比的几个参数取值，由于命令行操作的局限性，我们在此处不支持，如果需要使用，可以通过创建python脚本来调用statistics tools实现。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "99wH31s-lxg2"
   },
   "source": [
    "接下来，我们对上面运行的实验进行分析。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kJG87jyMl3Rs"
   },
   "outputs": [],
   "source": [
    "!omnisafe analyze-grid --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qy3iI_TT9jAP"
   },
   "source": [
    "关于数据分析工具的具体用法在第一节当中作了更为详细的介绍。此处我们仅展示如何通过CLI调用。\n",
    "\n",
    "生成上述experiment grid实验当中，在不同环境上PPO和PolicyGradient的性能对比图。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VonxbBLy9iXT"
   },
   "outputs": [],
   "source": [
    "!omnisafe analyze-grid ./benchmark/test_grid algo --compare-num 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hjVsp-UXmBrG"
   },
   "source": [
    "### Explore\n",
    "\n",
    "OmniSafe是一个年轻的，不断成长的库，我们会持续完善库的功能性并且优化用户体验，本节旨在覆盖OmniSafe中CLI工具的核心用法。您可以对新的交互方式和功能保持期待，同时，如果您在使用过程中有更好的见解，欢迎联系我们，我们将会认真听取您的建议。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fY4zSf60m6Bi"
   },
   "source": [
    "通过`--help`，您可以保持对OmniSafe的探索。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YwfeK-DOm-6J"
   },
   "outputs": [],
   "source": [
    "!omnisafe --help"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyMg768kwqjzNmWvNC1w3ryQ",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
